Automatic detection and correction of non-red eye flash defects

ABSTRACT

A technique for detecting large and small non-red eye flash defects in an image is disclosed. The method comprises selecting pixels of the image which have a luminance above a threshold value and labeling neighboring selected pixels as luminous regions. A number of geometrical filters are applied to the luminous regions to remove false candidate luminous regions.

PRIORITY

This application claims the benefit of priority under 35 USC §119 toU.S. provisional patent application No. 60/773,714, filed Feb. 14, 2006.

BACKGROUND

1. Field of the Invention

The present invention relates to a system and method for automaticallydetecting and correcting non red-eye flash defects in an image, and inparticular, white-eye flash defects.

2. Description of the Related Art

Published PCT patent application no. WO 03/071484 A1 to Pixology,discloses a variety of techniques for red-eye detection and correctionin digital images. In particular, Pixology discloses detecting “glint”of a red-eye defect and then analyzing the surrounding region todetermine the full extent of the eye defect.

U.S. Pat. No. 6,873,743 to Steinberg discloses a similar technique whereinitial image segmentation is based on both a red chrominance componentand a luminance component.

White-eye defects (white eyes) do not present the red hue of the morecommon red eye defects. White eye occurs more rarely but under the sameconditions as red eye, i.e. pictures taken with a flash in poorillumination conditions. In some cases, white eyes appear slightlygolden by acquiring a yellowish hue.

There are two main types of white-eye, small and large. Small whiteeyes, as illustrated at reference 10 in FIG. 1, appear on far distantsubjects. They resemble luminous dots and information in theirneighborhood about other facial features is poor and thereforeunreliable. Large white eyes 20 as illustrated in FIG. 2, are very welldefined and one can rely on information around them. In general, a whiteeye is large if it occupies a region including more than 150 pixels (fora 1600×1200 pixel image).

It is desired to have a technique for detecting and/or correcting whiteeye defects.

SUMMARY OF THE INVENTION

A method is provided for detecting non-red eye flash defects in animage. One or more luminous regions are defined in said image. Eachregion has at least one pixel having luminance above a luminancethreshold value and a redness below a red threshold value. At least onefilter is applied to a region corresponding to each luminous region. Theroundness of a region corresponding to each luminous region iscalculated. In accordance with the filtering and the roundness, it isdetermined whether the region corresponds to a non-red eye flash defect.

The defining may include selecting pixels of the image which have aluminance above a luminance threshold value and a redness below a redthreshold value, and grouping neighboring selected pixels into the oneor more luminous regions.

The method may further include correcting the non-red eye flash defect.One or more pixels of a detected defect region may be darkened. It maybe determined not to darken pixels within a detected defect regionhaving an intensity value greater than a threshold. The correcting, foreach pixel of a detected defect region, may include setting itsintensity value to an intensity value substantially equal to an averageof the intensity values of pixels on a boundary of the defect region. Anaveraging filter may be applied to the region after the correcting.

The at least one filter may include any of a size filter for determiningif said region is greater than a size expected for said non-red flashdefect, a filter for adding pixels to a luminous region located with theluminous region and which have luminance below the luminance thresholdvalue or a redness above the red threshold value, a skin filter fordetermining if the region is located within a region of an imagecharacteristic of skin, or a face filter for determining if the regionis located within a region of an image characteristic of a face, or anycombination thereof.

The roundness calculation may be performed by a filter to determine ifthe region is a non-red eye flash defect.

For each luminous region, a corresponding aggregated region may bedetermined by determining a seed pixel for the aggregated region withina luminous region, and iteratively adding non-valley neighbouring pixelsto the aggregated region until no non-valley neighboring pixels adjacentto the aggregated region remain. The region corresponding to eachluminous region may be the aggregated region corresponding to theluminous region. Contrast may be calculated for an aggregated region bycomputing a ratio of the average intensity values of pixels on aboundary of said aggregated region to the intensity value of the seedpixel. It may be determined whether each aggregated region has ayellowness above a yellow threshold value. An average saturation may becalculated for each aggregated region and it may be determined whetherthe saturation exceeds a threshold value.

The at least one filter may be applied to the luminous region. Anintensity gradient may be calculated for each luminous region. A Houghtransform may be performed on each intensity gradient. A mostrepresentative circle on each transformed region may be determined, andeach circle verified.

A digital image processing device is also provided that is operable todetect non-red eye flash defects in an image, and which includes acontroller that is arranged to define one or more luminous regions inthe image. Each region has at least one pixel with a luminance above aluminance threshold value and a redness below a red threshold value. Atleast one filter is applied to a region corresponding to each luminousregion. The roundness of a region corresponding to each luminous regionis calculated. In accordance with the filtering and the roundness, it isdetermined whether the region corresponds to a non-red eye flash defect.

The device may be a digital camera or camera phone, a general purpose,portable or hand-held computer, a printer or a digital scanner, or anycombination thereof.

A further method is provided for correcting a white eye defect in adigital image. The method includes acquiring a digital image, anddetermining a luminance of pixels within the digital image. Those pixelshaving a luminance above a certain threshold are selected as candidateregions for correction of a white eye defect. The selected pixels arefiltered, and white eye defect is corrected for non-filtered pixelsamong the candidate regions.

The filtering may include geometrical filtering of pixels based on asize or shape or both of a selected pixel region. A selected pixelregion may be above a threshold size, for example. Skin tone or humanface filtering of pixels may be based on neighboring pixels to aselected pixel region not having a skin tone or other human facecharacteristic.

Roundness may be calculated for a selected pixel region, which may becorrected if the roundness does not exceed a certain threshold value ofroundness.

The correcting may include calculating a contrast of a selected pixelregion, which may be corrected if it does not exceed a certain thresholdvalue of contrast.

The filtering may include checking whether an average saturation of aselected pixel region exceeds a certain threshold saturation, andcorrecting the selected pixel region only if the threshold is exceeded.

A bright pixel may be selected as a seed pixel. A candidate region maybe determined by aggregating outwardly from the seed pixel to combinethose pixels that are not valley points with the seed pixel as anaggregated region until a minimum number of non-valley neighbors areleft or a threshold size is reached, or a combination thereof. Theminimum number may be zero. Intensities of points in the aggregatedregion may be set to an average intensity of valley points delimitingthe region. The aggregated region may be smoothed.

The filtering may include determining and analyzing edges of candidateregions. An intensity gradient may be computed for one or more candidateregions. The one or more candidate regions having intensity gradientcomputed may be limited to include only candidate regions having aminimum size. A Hough transformation may be performed on the intensitygradient image corresponding to each candidate region. Candidate circlesproduced by the Hough transformation may be determined, and thecandidate region may be filtered and not corrected when the seed pixelis not included in the candidate circle or the average gradient alongthe circle is below a threshold, or both.

A candidate region that is merely a glint may be filtered.

The method may also include detecting and correcting a red eye defectwithin the digital image.

One or more digital storage devices are also provided having executableprogram code embodied therein for programming one or more processors toperform a method as described herein.

BRIEF DESCRIPTION OF THE DRAWINGS Statement Regarding Color Drawings

-   -   This patent application file contains at least one drawing        executed in color. Copies of this patent or patent application        publication with color drawing(s) will be provided by the Office        upon request and payment of the necessary fee.

Embodiments will now be described, by way of example, with reference tothe accompanying drawings, in which:

FIGS. 1( a) and (1 b) (hereinafter “FIG. 1”) illustrate an image withsmall white-eye defects;

FIGS. 2( a) and (2 b) (hereinafter “FIG. 2”) illustrate an image with alarge white-eye defect;

FIG. 3 depicts a flow diagram of the automatic detection and correctionof small white-eye defects;

FIG. 4 depicts a flow diagram of the automatic detection and correctionof large white-eye defects;

FIG. 5( a) illustrates a grey-level version of an image to be corrected;

FIG. 5( b) illustrates an edge-image of the image of FIG. 5( a) carriedout using a Sobel gradient; and

FIG. 5( c) illustrates a most representative circle of the image of FIG.5( b) as produced using the Hough Transform.

DETAILED DESCRIPTION OF THE EMBODIMENTS

A method is provided for automatic detection and correction of smallwhite eyes. A flowchart illustrating one embodiment is shown in FIG. 3.In this embodiment, an eye defect is said to be white or golden if it isbright, for example, in Lab color space the local average luminance I ishigher than 100, and is not too saturated, for example, in Lab colorspace, the absolute value of a and b parameters does not exceed 15.

Initially, the luminance of each pixel of an acquired image 250 to becorrected is determined and a selection of all the pixels whoseluminance is larger than a threshold value is made, 300. In thepreferred embodiment, the acquired image is in RGB space and theintensity is calculated as I=max[R,G] and the intensity threshold valueis 220. Also, to avoid highly-saturated colors (such as pure red or puregreen) the saturation computed as abs(R—G) is compared to a threshold of35, and discarded if higher. As such, only high-luminance pixels areretained, which provide seeds for a future region growing procedure.

In alternative implementations, the formula for luminance can be takenas the Y value for an image in YCbCr space. However it will beappreciated that luminance can be taken as the L value for an image inCIE-Lab space or indeed any other suitable measure can be employed.

The selected pixels are then labeled 310. This involves identifyingselected pixels neighboring other selected pixels and labeling them asluminous regions of connected selected pixels.

These luminous regions are then subjected to a plurality of geometricalfilters 320 in order to remove luminous regions, which are not suitablecandidates for white eyes.

In the preferred embodiment, the regions first pass through a sizefilter 321 for removing regions whose size is greater than an upperlimit. The upper limit is dependent on the size of the image, and in anembodiment, the upper limit is 100 pixels for a 2 megapixel image.

Filtered regions then pass through a shape filter 322, which removes allsuitably sized luminous regions, which are not deemed round enough. Theroundness of the luminous regions is assessed by comparing the ratio ofthe two variances along the two principal axes with a given threshold.Regions such as those comprising less than approximately 5-10 pixels,are exempt from passing through the shape filter, as for such smallregions, shape is irrelevant.

Filling factor 323 is a process that removes empty regions bounded bythe luminous regions if certain criteria are met. In the preferredembodiment, the ratio of the area of luminous region to the area of thebounded empty region is determined and if this ratio is below a certainthreshold, for example, 0.5 in one embodiment, the luminous region isremoved.

The remaining luminous regions are finally passed through a skin filter324 and a face filter, 325 to prevent white spots being mis-detected aswhite eyes based on the fact that they neighbor something that is notcharacteristic of the human face or skin color.

Skin around white-eye tends to be under-illuminated and turn slightlyreddish. A wide palette of skin prototypes is maintained for comparisonwith the pixels of the luminous regions. For each luminous region, theratio of pixels, characteristic to the human skin, to pixels, which arenot characteristic to the human skin, in a bounding box, is computed andcompared to a threshold value. In the preferred embodiment, thethreshold is quite restrictive at 85-90%.

Similarly, a wide palette of possible face colors is maintained forcomparison with the pixels of the luminous regions. For each luminousregion, the ratio of pixels, characteristic to the human face, topixels, which are not characteristic to the human face, in a boundingbox, is computed and compared to a threshold value. In the preferredembodiment, the threshold is quite restrictive at 85-90%. If the imposedpercentage is met or exceeded, the region proceeds to the step of regiongrowing, 330.

Region growing 330 begins by selecting the brightest pixel of eachsuccessfully filtered luminous region as a seed. Each neighbor of theseed is examined to determine whether or not it is a valley point. Avalley point is a pixel that has at least two neighboring pixels withhigher intensity values, located on both sides of the given pixel in oneof its four main directions (horizontal, vertical and its twodiagonals). As illustrated below in table 1, the central pixel withintensity 99 is a valley point because it has two neighbors in a givendirection that both have greater intensity values. Table 2 illustrates acentral pixel, 99, which is not a valley point because there is nosaddle configuration on one of the four main directions.

TABLE 1 100 98 103 70 99 104 104 105 98

TABLE 2 100 98 103 70 99 104 98 105 98

Starting from the seed, an aggregation process examines the seed pixel'sneighbors and adds these to the aggregated region provided that they arenot valley points. This examination and aggregation process continuesuntil there are no non-valley neighbors left unchecked or until amaximum threshold size is reached. If a maximum threshold size isreached, the region is deemed not to be a white eye and no furthertesting is carried out on this region.

The outcome of this stage is a number of aggregated regions, which havebeen grown from the brightest points of each previously defined andfiltered luminous region, and aggregated according to the valley pointalgorithm. It will be seem however that in alternative implementations,aggregation could take place before filtering and so the filters 320could be applied to aggregated regions rather than luminous regions.

A number of computations are then carried out on these aggregatedregions 340.

The roundness of the aggregated region is calculated 341 asR=perimeter2/(4.π.Area), where R≧1. R=1 for the perfect circle, and thusthe larger the R value, the more elongated the shape. White-eyes shouldbe round and so must be characterized by a value of R that does notexceed a certain threshold value. In the preferred embodiment, thethreshold value for R is a function of eye's size. Thus we expect an eyeto be rounder as its size increases (the smaller the eye, the poorer theapproximation of its shape by a circle, and the less accurate the circlerepresentation in the discrete plane). Three thresholds are used in thepreferred embodiment (for a 2 megapixel image—these will scale linearlyfor larger/smaller image sizes):

R=1.1 for large eye (i.e., size between 65 and 100 pixels—for a 2megapixel image);

R=1.3 for medium-sized eye (size between 25 and 65 pixels); and

R=1.42 for small eyes (size less than 25 pixels).

The contrast of the aggregated regions is then computed 342 as the ratioof the average intensity of the valley points delimiting the aggregatedregion to the maximum intensity value inside the region, i.e. theintensity of the brightest seed point from step 330. As small white eyesoccur normally in low illumination conditions, the contrast should behigh.

Most of the small white-eyes have a yellowish hue meaning that they haveat least some pixels characterized by high values of the b component inLab space. Therefore the maximum value of b, bmax, is a gooddiscriminator between actual white-eyes and for instance, eye glints orother point-like luminous reflections.

In one embodiment, the pixels being processed are in RGB color space. Inorder to obtain a value for the b component, the aggregated regions aretransformed from RGB color space to Lab color space.

The maximum value of the b component, b_(max), in Lab colour space isthen calculated and compared with a threshold, b_(threshold), at 343. Ifb_(max)≧b_(threshold), the average saturation in the region is thencomputed at 344. Otherwise, the aggregated region is deemed not to bewhite-eye.

The average saturation in the aggregated region is computed as:S=√{square root over ((a2+b2))} 344. White-eyes are more colored thanother regions and as such the region's average saturation must exceed athreshold in order for a candidate region to be declared white-eye at350. Aggregated regions passing the tests outlined above are labeledwhite-eyes and undergo a correction procedure 399 according to thepreferred embodiment for the present invention.

The correction procedure comprises setting the intensity I in LAB space,of the aggregated region's points to the average intensity of the valleypoints delimiting the region as used in the contrast calculation at 342.In the preferred embodiment, the whole aggregated region is thensmoothed by applying a 3×3 averaging filter.

According to a further embodiment, there is provided a method forautomatic detection and correction of large white eyes, as depicted inthe flowchart of FIG. 4. The main characteristics of large white eyes isthat by being very well defined, their shape is round and they are wellseparated from the iris.

Referring to FIG. 4, it can be seen that the first five stages of thelarge white-eye automatic detection process, thresholding 400, labeling,410, size filter 430, shape filter 440 and filling factor 450, areidentical to those of the small white-eye automatic detection process asdescribed above. However, it will be seen that the threshold applied inthe size filter 430 will be larger than for the step 322 and thatdifferent parameters may also be required for the other stages.

Nonetheless, once the luminous regions have passed through thegeometrical filters 420, the next steps determine and analyze the edgesof the suspected large white-eyes.

First, an intensity gradient of each luminous region is computed 460.The gradient is calculated from a grey-scale version of each luminousregion as depicted in FIG. 5( a). Gradient is any function that has ahigh response at points where image variations are great. Conversely,the response of the gradient is low in uniform areas. In the preferredembodiment, the gradient is computed by linear filtering with twokernels, one for the horizontal gradient, Gx, and one for the verticalgradient, Gy. The modulus of the gradient is then computed asG=sqrt(Gx2+Gy2) and is further thresholded to obtain edge points andproduce a binary edge-image as depicted in FIG. 5( b). In the preferredembodiment, step 460 is carried out using a simple Sobel gradient.However it will be appreciated that any gradient function such asPrewitt or Canny may be used.

Once the edges of the suspected large white-eye regions have beendetermined, a Hough Transform is performed on each gradient image, 470.A Hough Transform detects shapes that can be parameterized, for example,lines, circles, ellipses etc and is applied to binary images, usuallycomputed as edge maps from intensity images. The Hough Transform isbased on an alternative space to that of the image, called accumulatorspace. Each point (x,y) in the original image contributes to all pointsin the accumulator space, in this case, corresponding to the possiblecircles that may be formed to contain the (x,y) point. Thus, all pointscorresponding to an existing circle in the original edge-image will allcontribute to that point in the accumulator space corresponding to thatparticular circle.

Next, the most representative circle as produced by the Hough Transformmust be detected for each region, 480. This step comprises inspectingthe points in the Hough accumulator space, which have a significantvalue. This value is dependent on the number of points in the originaledge image, which contribute to each point in the accumulator space. Ifno representative circle is found, there is deemed to be no large whiteeye present in that region of the image.

However, if a high value point is found, then the corresponding circlein the original image is checked and a verification of the circle 490 iscarried out.

This involves checking for example whether the most representativecircle encircles the original seed point for the luminous region and/orwhether the average gradient along the circle exceeds a threshold.

If a circle of a luminous region is verified, the region is corrected,499, by darkening the pixels in the interior of the circle. In thepreferred embodiment, the intensity of the pixels is set to 50 and anaveraging filter is applied.

Preferably, however, the correction also takes into account thepossibility of the luminous region including a glint, which should notbe darkened. In RGB space, glint candidates are selected as highluminance pixels (min(R, G)>=220 and max(R, G)==255). If a very round(both in aspect ratio and elongation), luminous, and desaturated regionis found within the interior of a luminous region, its pixels areremoved from the luminous region pixels to be corrected.

In the case where further eye-color information is available, forexample in the case where person-recognition procedures are availablewith a database of previously captured images, the additional colorinformation stored with that person's information in the database can beadvantageously incorporated into the correction of both large and smallwhite-eye.

In methods that may be performed according to preferred embodimentsherein and that may have been described above and/or claimed below, theoperations have been described in selected typographical sequences.However, the sequences have been selected and so ordered fortypographical convenience and are not intended to imply any particularorder for performing the operations.

In addition, all references cited above herein, in addition to thebackground and summary of the invention sections, are herebyincorporated by reference into the detailed description of the preferredembodiments as disclosing alternative embodiments and components. Thefollowing references are also incorporated by reference:

U.S. patent application Ser. Nos. 11/462,035, 11/282,955, and

United States published patent applications nos. 2002/0136450,2005/0047655, 2004/0184670, 2004/0240747, 2005/0047656, 2005/0041121,2005-0140801, 2005-0031224; and

U.S. Pat. No. 6,407,777.

The present invention is not limited to the embodiments describedherein, which may be amended or modified without departing from thescope of the present invention.

1: A method for detecting non-red eye flash defects in an image, saidmethod comprising: (a) defining one or more luminous regions in saidimage, each region having at least one pixel having luminance above aluminance threshold value and a redness below a red threshold value; (b)applying at least one filter to a region corresponding to each luminousregion; (c) calculating the roundness of a region corresponding to eachluminous region; and (d) in accordance with said filtering and saidroundness, determining whether said region corresponds to a non-red eyeflash defect. 2: A method according to claim 1, wherein said definingcomprises: (i) selecting pixels of the image which have a luminanceabove a luminance threshold value and a redness below a red thresholdvalue; and (ii) grouping neighboring selected pixels into said one ormore luminous regions. 3: The method according to claim 1, furthercomprising correcting said non-red eye flash defect. 4: The methodaccording to claim 1, wherein said at least one filter comprises; (i) asize filter for determining if said region is greater than a sizeexpected for said non-red flash defect; (ii) a filter for adding pixelsto a luminous region located with said luminous region and which haveluminance below said luminance threshold value or a redness above saidred threshold value; (iii) a skin filter for determining if said regionis located within a region of an image characteristic of skin; or (iv) aface filter for determining if said region is located within a region ofan image characteristic of a face, or combinations thereof. 5: Themethod of claim 1, wherein calculating said roundness is performed bysaid at least one filter to determine if said region is a non-red flashdefect. 6: The method according claim 1, further comprising, for eachluminous region, determining a corresponding aggregated region by: (i)determining a seed pixel for said aggregated region within a luminousregion; and (ii) iteratively adding non-valley neighbouring pixels tosaid aggregated region until no non-valley neighboring pixels adjacentto said aggregated region remain. 7: The method according to claim 1,further comprising applying said at least one filter to said luminousregion. 8: The method according to claim 1, further comprisingcalculating an intensity gradient of each luminous region. 9: A digitalimage processing device operable to detect non-red flash eye defects inan image, and comprising a controller arranged to: (a) define one ormore luminous regions in said image, each region having at least onepixel having luminance above a luminance threshold value and a rednessbelow a red threshold value; (b) apply at least one filter to a regioncorresponding to each luminous region; (c) calculate the roundness of aregion corresponding to each luminous region; and (d) in accordance withsaid filtering and said roundness, determine whether said regioncorresponds to a non-red flash eye defect. 10: A device according toclaim 9, comprising: a digital camera or camera phone, a generalpurpose, portable or hand-held computer, a printer or a digital scanner,or combinations thereof. 11: One or more digital storage devices havingexecutable program code embodied thereon for programming one or moreprocessors to perform a method of correcting a white eye defect in adigital image, the method comprising: (a) acquiring a digital image; (b)determining a luminance of pixels within the digital image; (c)selecting those pixels having a luminance above a certain threshold ascandidate regions for correction of a white eye defect; (d) filteringthe selected pixels; and (e) correcting the white eye defect fornon-filtered pixels among the selected pixels. 12: The one or morestorage devices of claim 11, wherein the filtering comprises geometricalfiltering of pixels based on a size or shape or both of a selected pixelregion. 13: The one or more storage devices of claim 11, wherein thefiltering comprises geometrical filtering of pixels based on a size of aselected pixel region being above a threshold size. 14: The one or morestorage devices of claim 11, wherein the correcting comprisingcalculating a roundness of a selected pixel region, and correcting theroundness if it does not exceed a certain threshold value of roundness.15: The one or more storage devices of claim 11, wherein the filteringcomprises checking whether an average saturation of a selected pixelregion exceeds a certain threshold saturation, and correcting theselected pixel region only if the threshold is exceeded. 16: The one ormore storage devices of claim 11, the method further comprising: (i)selecting a bright pixel as a seed pixel; and (ii) aggregating outwardlyfrom the seed pixel to combine those pixels that are not valley pointswith the seed pixel as an aggregated region until a minimum number ofnon-valley neighbors are left or a threshold size is reached, or acombination thereof. 17: The one or more storage devices of claim 16,the method further comprising smoothing the aggregated region. 18: Theone or more storage devices of claim 16, the method further comprisingcomputing an intensity gradient for one or more candidate regions. 19:The one or more storage devices of claim 11, the method furthercomprising filtering a candidate region that comprises merely a glint.20: The one or more storage devices of claim 11, the method furthercomprising detecting and correcting a red eye defect within the digitalimage.